Skip to main content

Measurement: Event-Based Data Sufficiency & Disqualification Guidelines

Overview

This document includes guidelines that define the minimum data and model quality requirements for accurate modeling and measurement of event-based programs (e.g., demand response, load shifting). Meters that do not meet these standards may be assigned a savings estimate based on the measured population and/or may be excluded from reporting to maintain analytical integrity.

Consumption Data Definitions

The consumption data types listed throughout the document are defined below for context.

Hourly:

  • Hourly data typically provides 1 value per meter for each hour’s total usage
  • Default logic checks data sufficiency at the hourly level

Sub-hourly:

  • Sub-hourly data typically provides 1 value per meter for each 15-minute interval, aligned with the quarter-hour, or at a more granular level (e.g., 5 minutes, 1 minute, etc.)
  • Default logic rolls up more granular consumption data to 15-minute values
  • Default logic checks data sufficiency at the 15-minute interval level

Training Period Requirements

Data Sufficiency Requirements​

The standard training period to measure an event is 45 days prior to the week in which the event is called. The following criteria apply to the data in this period.

  • At least 35 days within the 45-day training period must contain data for 90% or more of expected readings for each of meter, temperature, and solar data
  • Each hour of the week must have meter data from at least three different days
  • The meter data must have sufficient variability: at least 10% of the data points must have unique values

Meters that fail these requirements will be disqualified from measurement, and the reason will be documented. For meters with solar or battery installation dates during the training period, all data prior to the transition will be treated as missing for model training and data sufficiency.

Model Fit Requirements​

Meters must have consumption patterns that allow for the generation of a model with a reasonable fit.

Recurve’s platform computes a model’s:

  • Coefficient of Variation of Root Mean Squared Error (CVRMSE),
  • And its Percentile Normalized Root Mean Squared Error (PNRMSE), defined as the RMSE divided by the interquartile range of a meter’s interval consumption

These metrics are utilized to disqualify meters exceeding certain thresholds

  • CVRMSE between 0 and 1.4 OR PNRMSE < 2.2

Meters that fail these thresholds will be disqualified from measurement due to poor model fit.

Comparison Pool Requirements​

Each event participant must have a sufficiently large set of non-participant comparison-group meters that meet all of the training period requirements and the measurement period data sufficiency requirements. Participant meters for which a suitable comparison group cannot be constructed will be disqualified from measurement, and the reason will be documented.

Measurement Period Requirements

Data Sufficiency Requirements​

The measurement period consists of the 24 hours ending at midnight on the day the event was called. Within this period, the following data sufficiency requirements must be met.

  • All event intervals must have valid and complete meter, temperature, and solar data.
  • Non-event intervals must contain data for 90% or more of expected readings for meter, temperature, and solar data, as well as the joint dataset including all three sources

Dispatched meters that fail these reporting period thresholds will be disqualified from measurement, and the reason will be documented. For meters with solar or battery installation dates within the measurement period, all data subsequent to the transition will be treated as missing for measurement and data sufficiency purposes.

Impact Outlier Thresholds​

Very large load reductions (or increases) relative to the counterfactual often indicate that a building’s consumption changed in ways that are unassociated with an event call. To maintain the integrity of measured results, meters with load impacts that are very large outliers (fractional load impacts more than ~5 standard deviations from the median for the dispatched population) will be disqualified from measurement, and the reason will be documented.

Dual Participation​

Meters that are dispatched in more than one event on the same calendar day (e.g., due to dual program enrollment) will be disqualified from measurement, and the reason will be documented.

Disqualification Treatment

Meters that are disqualified from measurement for any of the reasons above will still receive a performance value. In these cases, a meter will be assigned a value aligned with the measured performance of participating meters in the same measurement grouping.

For more details on Recurve's approach to disqualification and savings assignment, see this document.